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XBSTBACT , : 

" Evaluation procedures for , the syateiatic lea^urement 
of Extension^ s'total lipict or ef f ectlvesese In rtral Shaifano County, 
Wisconsin, Involved developing compreh€nalve descriptlena of 
Extension programme from 196^ to 1975 to list typea c£>fLCtlvltles an4 
offerings, topics, attend^fcfsi and cyervievi ci effdwts *in progrmm 
arefs Ilka agriculture, home economics, and Iralned picfesslonal 

interviewers contacted by telephone 238 crusty cr cotifflunlty leaders 
and a random sample of 1192 cfunty reaideati 18 yeara or older. 
Fourteen general conclusions on methodology ar# presented and * 
discussed for the ^se of others who are cctfslddrln^ a similar 
evaluation, Them inqludei (1) the concept of »lmpact" ^ the levels of 
evidence hierarchy, and the general teneflti model help crganize an ^ 
evaluation; (2) different reaaons for tctal frog lam' evaluation x 
influence design and data sources; (3) thorough docuttentatlon of the 
Extension program. is necessary*^ (U) evaluatlcn should focus on recent 
(3-5 Taars) efforta when trying to assess iipact dver time; anff (5) 
exact replication of this entire evaluation is unneceasarn 
suggestions ap€ given for the most effective areas of evaluation. 
(DS) . ^ 
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Laverne Forest wob county agricultural agent in Minnesota for eight yearw -prior to ^kuming hw 
current role. as ^valuatibn SpecidliBt in Program and Staff Development, University of Wisconsin^ 
Extension, He is an associaM prhfesMorin Contitming and Vocational Education on the Madison 
jdampm. .^-t^— ^ K s ^ 

Mary Marshall i$ research assmtant with ^ogratn and Staff Development, and doctoral candidate 
in continuing education—evaluation. She wcm Extension area youth specialist in Missouri for ^ 
nine yearn* , ■ \ 



Cpnalusloni v 

' '1. The concept bf "impict" ^6 liveli-of-ividence hierarchy, and thfe^ntral 
binefiti modiLhilp organiii^an ivdlua^^ , ^ 

2. Diffgrint faaforis far totakprogram tmpfct eviluation Influence design ind/ 

data sources, ■■' ' J ' ' - 

5^ Thorough documthtatlon of the Extension pragra 
\ ■■ 4. Gfnerill^fng iubstudies'of impact in ip^if Ic program areai and audienqes 
' is^possiblt thKDughi ntenile wing 'one large rti^ ^ - 

Stitewlde^br district sampling can increase the survey's general value tf \ 
specific coun^ resulm are of less concern ' ' / \ 

6, Respondent' perCBptions of Impact are valid alternativei to em^Tical \ / 
ol^arvations of actual befiaviors* ' V j 

. 7. Retrbipective evaluations identify major progrirri relitlonships and ad^tive 
effect beyond those implied in InltiaLprogram ob]ectlves. \ f 
8* Eviluation should focus on recent {3-5 year) efforts when trying to a^^si 

impact over time. . : 

9. Exact repliqation of this entire, evaluation Is unnecessary^ — — ^^.^^ 

10. A program evaluation of this magnitude requires an unusually li^e person- 
nel investment before, during, and after data collectron, ^ 

1 1 , Complex rtatistlcal analysis is not necessary for optimum utility and under- 
standing of evidence. , . ^ 

12. Understanding data and deriving conclusions ari as crucial as design and 
, data col lection* 

13, Rfciplents of evaluation data must identify their criteria for Interpretation 
and riference if data are to become meaningful. 

' 14y A variety of timely,, short, simple, attractively visual lied tachnlques to 
/ comm\Jnlcate evidence gete acceptance. 

/What Next? , \ 

Additional Materials 
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How can wt test datermini who his eontaet With ixtension? How em wt get 
ptopli's reactions to Extension? What kmds of data are appropriate in total 
Impact eviluations? What prooaduris will halp data, to bi unda^tood and ustd? 
T\\%m wtre somt of the rnajor questions of the Shawano Coun^ evaluation. 

The Shawano County Evaluation Proj®:t has three main purposes: 

1. To systematically measure Extension's total impact or effectiviness in one 
county, ^ " - 

2. To develop evaluation procedures, 

3* To ixplore how evaluation data are used, . 

This report, as Part 2 in a series, summarizes key conglusions and Ideas about 
the procedjires used. We hope these ideas will help you dttermlne whether some 
of the procedures u^d could fit your Extension program or one you are familiar 
with or responsible for The usefulne^ of the Ideas will pFobably be more clear if 
you have a situation in mind as you read. ' \ ' ■ " 

Evaluation is def!ned.a$ rpaklng judgments about Extension's total effom at the 
county %vel by comparing actual contacts, reactions, etc, (data) with descriptions 
of what is desired (criteria). Evaluation is not seen as measurement, or as proof 
of effect, or as determining the attainment of educational objectryes. 



&ifnmary of Major Steps and Tasks 

1 , Selection of County, A cooperative agreement was made between the district 
dlrector/state Extension director, evaluation specialiste, county Extension 
staff, and tiie county Extension committee. Selection was subjectively based 
on expected cooperation, diversity of program, diversity of clientele, and scope 
of program, 

^ 2. Building Support During early stages of the project a lot of time was given 
to Informing all Extension administrators and program leaders, Shawano 
County Extension staff, and local leaders of itie project They increased 
their understanding of it, gave their approval, and also made many suggestiohs. 
State and national advisory committees were also formed for these purposes, 
3. Staffing and Resources. The county staff was not exjpected to carry the 
burden on the evaluation tasks of data gathering, interpretation, and commu- 
nication of results. In additiort to the project codireciors, a half-time research 
distant was hired for the duration of the project Two people from Shawano 
Coun^ were hired ad hoc to do local data gathering. Telephone interviewing 
was subcontracted to a professional survey office. 
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*4. Prqgrahi Inventories/ ad hoc locirstaff vyorking with project codirictors 
; daveloptd coniprehinsive descriptions of all Extinsion programs in the county 
from 1966-1975, These written discriptive summaries included typis of ; 
aqtivities and offej'ings, topics, attendance by various cliantilfe, and overvisws 
of effort in all program areas like agriculture/homs econormici and 
. Thtsi summariis not only were part of ivaluation, but defined the limits of 
\ the program to be considered-in survsys, \ * 

5. Think Groups, Small groups of administrators, prograrn Ipaderi, specialists, 
district directors, and Shawano agents reviewed the program inventoriii.^ 
Along^/vith advisory' committees/ these think groups made suggestions about 
the type of evaluatlve^dala they would find useful and how data^ught to be 
gathered. This activlw was intended to increase the jikelihood of eventual 
data usage. . ^ 

6. Surveys, Two groups of Shawano citizensjwere identified and^ interviewed by 
telephone in earty. 1976 by trained profepionakinterviewers, 

' Leader Sun/ey/ 238 people (or about 85% of those identified) who held an 
elected o^ appointed office or were considered leaders by people kngwladge- 
able about county and community affairs were interviewed^- 
Generdf^ Public Survey:^ A random sample of 1,192 reiidents, 18 yean or 
oider, VN^ere interviewed. Answers to an early question separated those who 
haS considerable contact with Extension from those who had little or none. 
The former group was asked many more questions than the latter as inter- 
viewers used different irateryiew schedules, 

7. Preliminary, Data Summaries, The sums of various responses and percentagei 
from sun/ey data were prepared for immediate iharing with mtmheri of 
tinink groups, advisory committees, and other Extension staff int^ested'in 

r early results. ' , ^ 

^ 8. Standard Setting, As sun/ey summaries were shared via meetings and mail, a 
procedure was used thereby recfpients were asked to state certain minimum 
data levels they woui^ accept as indicative of program success. This proceduri 
evolved as a central Aspect of the entire project because of its power to gen- ^ 
erate interest, to help Extension staff put rraaning on data, and to form 
judgment on how good the county program was, 
9, Judgments and Conclusions, The Shawano County Extension stiff diseased 
* preliminary data and made judgments. Program leaders, administrators, and 
subject-matter specialists offered their interpretations and conclusions. The 
evaluation team struggled with reaching a consensus on major conclusions 
and implications, ahd eventually fhe first major report was published, 
10, Communication of^Results, Various written, visual, and verbal techniques 
were u^ed. Evaluation results were shared via the book Impact of Extension 
in Shawano Courity: ^^7, Conclusions andJmfilicafions/ many summary leaflets, 
and>!pore detailed summfiries, Overhead transparency presentations were n 
given k many meetlngs^y the district director, evaluation team and county 
Extension staff. Also, many one-to-one contacts were made.'^ " ^ 



11, Ev§luatlng the^EvBluation. Tht nature and txttnt of uiage of the evaluation ^ 
_ result by Extinsfon itaff prirparlJy In Wisconsin were detirrtiined/ These data, 
7 along witfi additional recommendations for what data art ustfuL comprisi 
Impact of Extension in Shawano 3, Usage and AppraisML 
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Conclusions 

The rf mainder of this secona maj^ report .presenti 14 gensral conclusions on^ 
methodoTogy for considerAion by others who are exploring the poisibllities of a 
similar evaluation^^'^rtslof It ^aders are also encouraged to review page 23 ^ 
which lists other available reference pieces. 

Discussion following each conclusion includes additional detail about proce- 
dures and some suggestions for further application. 



8 



V 



evidence hlerarel^) and the Qmetml 



s moaei neip <M*pinize an 



IMPACT ^ ■ ' V 

. - ■ \* ■ 

Impact is the amount of Extinsion'i contact with people and the Imprttilons 

Extension Itavsi on them. As the diptrict director who requested this evaluatifan 

says, Impact is the degree^to which Exteniion penetrates the county, communities, 

and individuals, and the extent to which the result Is positive. 

Impact is indicated by the number of people contacted by all of Extension, the 
frequency of those contacts, their resultant learning, their application of Ideas, 
gained frorn those contacts, their perceptions of benete from Extension, and 
their judgments of Ex tension, * w 

This project determined Extension's impact usiQg perceptions of different 
groups; t 

• Total county population 

• Specific audiences, such as farmers, 4- H families, the elderly, profe^ionals, , 
and Native Americans, ' ^ ' ^ ' ^ 

• Key cooperators or heavy users of Extension, ^ 

• Leaders, ^ - 

Beciuse impact Is an Idea that encompasses various types of results^ it Is a useful 
concept It pulled together results due to multiple contacts, such as a family 
might have thcough several kinds of programs. This assessment of alf accomplisK- 
ments within one evaluation gave insights as to the total value of a program, dif 
ferent from what would be bbtained by any single assessment, 

A 

The Shawano evaluation considered all of Extension's impact in the county as 
one. The following efforts were included in the Shawano County program that . 
was evaluated: ' . 

• Both agent and specialist activities, ^ - 

• Activities in and outside the county In which residents participated, 

• All of Wisconsin's 13 prograrpp areas. ^ - 

• All methods including ma^ media, group, and individual contacts, 

• Extension efforts with various groups and communities, 

• Repetitive contacts oyer a multiple-year period (15 years), 

.Agente were Initially surprised that the evaluation would include such a broad 
spectrum of efforts. Advantages soon^became apparent: Extension efforts witK 
certain commodity and other small groups could be shown more fully if combined; 
the value of input from statewide specialist and resources Into a county program 



could be shown; contacts with more than one program area by an Individual could- 
^ documentidj and comparative data on the effectiveness of mtthods such as 
bulletins, itate radio network, arid conferences.could hilp Extenii on decide hdw 
nriuch to invtit in each, ^ . '. 

LiViLS-OF-EVIDENCE HIERARCHY ) 

Bennett's hierarchy served as a ba|ic tool for organizing irhpwt types and for 
identifying ne^ed evidence.'" The hierarchy ';said'^ (1) identify Exfihgitin's 
staff and budget commitment over the Ifryear period, (2) identify the activities 
conducted or performed so as to know what survey follow-up was appropriate, 
(3) Identify the number of people reached by Extension, |4) identify people s 
reactions to Extension, (5) identify the amount of learning due to Extension, 
(6) Identify any application of new Ideas, and (7) identify any benefits due to 
applying ideas. j • . 
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TYPES OF 

impact' 
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/I INPUTS 



Structured interview queitipne (with liste of poislble risponses) ware developed 
for most of the itven levels, but the wide range of programs being evaluated 
meant that open-ended question! were alio needed. 

GtNERAL BENEFITS MODEL 

Forest provided ah additional framework for identifying, describing, analyzing, 
and iummariiing various impacts.^ Ueaders ai well ai tKe-general public were 
asked \A/hether they or their comtnunlties benefited from Exteniiorlin any of the 
following general ways: 

• Developing groups, government, and democratiS processes,/ ^ ^ \\ / 

• Developing irtdividual roles and abilities. 

• Improving health and safet/ facilities and practices, 

• Conserving and improving the natural environment 

• Securing economic improvement , 

• Expanding educational resources or opportunities. 



^Clauda'Bennett "Up the Hierarchy/' Jouma/ orExf^n^/on. XIII (March^April, 1975), 7-12. 
^Laverne B. Forest "Using Values to Identify Program JoumBl^EKtemQn, XI (Fall, 



1973), 24-34. 
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Ba^d on major value systerns ind riprisentative of long-tirm goals for humin 
endeavors, the-six typei,were used to categorize risponsti ihowtng how Extaniion 
tfforte In various and diverse ways had contributed toward cintral themes In 
pipple*s fives. ^ 

^ rtium, our e^periinces lid to several specific suggestions for future ''total pro= 
grafl'' evaluations: j 

' \ 

• Conceptuil frameworks are needed to design, carry out. Interpret, and com- 
municate total county program evaluation results* 

• The selicted frameworl* must encompass many thing^^^ 

gram areas, multiple-year efforts, multiple contacts, various outcomes, and 
various vafM systems of people. ^ ."" ^ 



a.TOf ^rait reascms for totol in^grani 
imiiact evaluation Influence design and 



sourees. 

The purpose of this evaluation was to get usable ''total impact" data (any and 
all kinds of contact) and resultant Impressions, Only secondarily was it concerned 
with contacts specific to certain subject matter or problems. The general evalua- . 
tion design thus fit all program areas, - ' \ 

Two general types of use for these total impact data ernerged as most releva 
Internal decision making and external accountability^. The county staff and st 
wide subjec^matter specialists were more likely to identify specific-evidences 
needed for thpir ovvmparticular program declslonV Administrators, in contast, 
were more likely to request data for reportrng and accountability purposes, ^ 

Following the logic in thesa divergent opinions, the evaluation team's task was 
to balance needs assessment and methods evaluation (for making internal decisions) 
with results evaluation (for accountability purposes). 

Our specific suggestions are: 

• Consider "total" impact evaluation such as this one if your general reason 
for evaluation Is accountability. ' 
. _ • Set priorities for usage early in an evaluati^. This will help channerideas 

into relevant questions and weed out extraneous "Gee whiz, that's interesting" 
type inquiries/ If program decision making is the focus of evaluation, inten- 
* lively involve all the relevant decision makers within programming units at an - 
early stage so they identify data they can use. They will need help in suggest- 
ing questions to, ask in surveys and in understanding evidence that comes froiji' 
surveys. 
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gh doeumentation ol the 

prograni is necessary. ' 

^iny sourcei of ditrseivtd to document Shqwa^ Office , 

racordi, annual reports/project files; cfrcular letters, agent vitae, riewi files, Wiscpn- 
sln EKtenslon managemfnt InformatioiV'syitsnf* {EMIS) data, and recordi of non- 
credit program offerings from state-based departments and specialists were all used * 
to develop Extension program summaries. Shawano agents were not involvid in 
Ihli dbcumentation process except to review informatiQn for accuracy and corn- 
pletiness. ^ ' 



The program summarlei were used by ipecial task forces called think g^pi^ 
Summaries indicated the scope and depth (^ programs and thui servad as a bails 
for deciding survey content and^focui. The summaries also documinted the extent 
of certain lon^range tfforte as they occurred" over several ytars--a story that. was ^ 
not as apparent whin looking at single-year efforts? The additive effects of con- 
tinuing sequential pregramming became much more evident thr@ugf\Jh is activity. 

^ Based on our expeciences, weTecommend you: 

• Prepartwa comprehensive surnmary(ies) for the program tPme period to be 
evaluated. ^ , , . 

• professional time, if possible, by hiring ad hoc people who are jomewhat 
familiar with Extension, . 

• Review not only county records; but those of other Extension units, 

• Use documentation for, planning data collection and evaluation, 

• Share documentation vyith potential users of evaluation data. 

^ - , . , ; - ' 

Generalizing substudtes of Impact In" ^ 
specific prc^ram areas and audiences Is 
possible through Interi^ewlng one large 
random sam^e. 

^In addition to questions on overall impact asked of all leaderi end citizens, -certain 
questions were asked of specific respondents depending on demographic character-' 
istici (such as occupation, race, association with 4pH, etc.) which allowed useful 
impact evaluations relative to farmers, 4-H families. Native Americahi, profe^ionals, 
and many others. . 

''Split interview icheduling" provided for gathering information on a variety of' 
topics without bufdening^every respondent Randomly assigning some interview 
questions allowed data to be treated as ^ough they came from all respondents* 
Several kinds of split scheduling were used; . 

First, several versions of the same interview were used with the general public. 
As an opening Interview question, respondents were asked whether they/d had "a 



lot/' ''somfe/' /'a Iittle>'' or ''ho'' contact with Extension Those angering, 
"some" or "a lot" were then random^ly^signed to orle of two lengthy interviews. 
Respondents saying they'd had "littte" or "no" contact were given a shortened / 
ihtervlew, This procedure minimized annoyance to people who'd had little or no 
contact, yet allowed results from each of the three types 6f interviews to fit to- 
gether. ' ■ * , 

• ■ fi; * . ^ ^ ^ 

For example, all respondents were asked if they'd used Extensioh-methods such 
as bulletihs, radio, television, jneetings, or events; HoWever, not every respondent 

was asked about the helpfulness of each method ^ 

■ \. % : * ' ■ . 

X| another example, all respondents weN asked to react to Extenston's person' ' 
nel and organization, butnot alLwere asked about the same ideas. Some were 
asked if Extension worth the^tax^money invested/yyhile otherl^ra asked if 
Extension was efficient in carrying put its work. , ^ * ' * . 

Second, interview subsections were used for speci^d general-public respondents. 

For instance, respondents answered questions about Homemaker Club experiences 

only if they'd had home economics program contact. Only farmers got questions 

fbout contact with Extension's agriculture-agribusiness educational program, 

' _ " ^1 

The impact of Extension through 4-H was determined by asking respondents . ^ 

additional questions if they^isaid they: (a) were current or past 4=H leaders; (b) had 

children in 4-H, eTther at the time of the interview or in the past 15 years; or' 

(c) Wfere themselves 4-H members at some tinrie. 

Third, leaders were randomly, assigned to answer questions on only one.of three 
major community resource development programs. One third of the leaders inter^ 
viewed (about 90 out of 272) were randomly selected ,to answer questions about . 
inGjustrial development efforts in the qount^, another third about outdoor recrea^ : 
tional resource development, aHci the other third about land use planning education 

^ In summary, sampling and interview design were important to data collection in 
the Shawano evaluation. Careful attention to drawing a sufficiently large random 
sample allowed analysit arid reports for many population segments, wfiere 200-500 
may have been large enough to generalize about the total adult populatmn, 1,200 
yvere neede'd to assuriujhat relevant subgroups would have sufficient numbers for 
' additional/analysis. Said another way, had it been necessary to get 100 Nafive 
An^^ericans or EFNEP parttcipants among respondents arid still keep the total sam^ 
pie representative of the^hole county, 3,000 Jnterviews would^have been required. 
In pur case, a compromise was made; given a limit to project resources. 

; One major exception in the evaluation to including a subsyrvey within a farger. 
survey was the leader survey, Sin§e county and community leaders wece felt to be 
a ke,y .gfou^^interview, a citizen sample would have to be too large to e'ncompass . 
a good leader sampling. Questions asked leaders,' were significantly different in many 
cases, so a separate survey /was used in this evaluation. The results were UQique^^and 
useful in thernserv#s, and others are encouraged to consider surveys of leaders as a 
iA/ay of gettrng useful impact evaluation data. C 



§• ^attewide o^^fstrict san^ling ?can 
incr^se the swyey^s general value it 
speelfilc county results are of l^s 
concern. 

^ Thivalue of generalizing evidence beyond a particular coun^ should^be confiid- 
. ered early in an evaluation, Is it better to invest in 100 interviews in, 10 rand^ly 
SBiected counties or in 1,000 inten/iews in ^ county? The answer depends on the 
intended use of the evaluation results. . ' ^ 

Basic to this decision is the structural makeup of the Extension program. ^When 
a strong "county^' system Js in eKistence, results may need to telj how strong a 
county prbgran^ is. However, because no typical' county exists, think about how 
useful the f^praings from ''a single county's program may be for specific deqisions in 
another county , * - ^ 

If regional, district, or area program planning predominates, couh^y]findings 
might have lesymeaning, • 

. J- ^ In summary, the type of sampling depends on what population the evidence is 
supposed to represent. Evaluation should consider geographic factors and organi- 
sation structure tha^ affect programs. . . ' 

6« Respondents^ perceptions of impact aire 
valid alternatives to xeniplrlcal obser- 
vations of actual beh^MVil 

* The Shawano County evaluation looked at Extension through local people's 
^ eyes, All data in the project=end results; reactions, contact— are perceptual. 



The reasons for emphasizing these data are: ' , 

• Perceptual data^re easier to collect than "hard evidence/' Scientific controls, 
observers, pre- and poitrmeasurements, arid othe/ canons of science are more 
difficult, if not impossible, to apply ingmulfiveaa multidiscipline, multimethod, 
multiaudience, and multistaff program evalUatiorTW 

• Voluntary adult programs depend more on percfeived value to participants 
^ or potenfial participants than on actual value. 

• P4rceptuaT data are less cpstl^, both in money and in irritation to respondents. 

• Perceptual data are more easily uhderstood. Feelings and testimonies of^ 
people are easily.,unders.tood,. wKLte some users may not understand how num- 
bers of actu^ftchanges made by people or iffkitutions affect their lives or 
reflect value of a program. 

• Perceptions allow respondents to review their experience retrospe^U^^^^d 
suggest major program sequences, interconnected events, and addTtive effects. 



% On the other hand, weaknesses in Shawano data are these: vague adjectives 
within response categories such as "a lot of contact with Extension "'are hard to 
interpret One person may think of "a lot'' as hundreds; another may think of tens. 
Users of data jlTant to know what adjectives, such as ''a lot/' mean. Wa do not 
recommend ufage of only vague adjective response categorieti^ 

In sum, whether actual" or per^eptuaKdata are collected defends on intended use, 
but dp not downgrade perceptual data because they fail tafneet the physi^aricien- 
tist's criteria for data, . / / 

Retro^Pfetlve evaluations identify 
mai^r plrograni relattonships and addi- 
tive effects beyond tliose implied in 
initial prograni objectives. 

- _ . _ . ' \ - . . f . 

Objectives state what Extension designs or intends, even when in consultation 

with learners. Perceptions are what people believe and feel, regardless of program 

intent. This evaluation, although limited by program summaries, waa not confined 

to determining if objectives were met ^ 

In this evaluation, local leaders and program participants looked'fnto their im- 
medifte past and Identified retrospectively the meaning that J^iension's efforts 
had for them and their communities. In this way, major p.^<fgrams were identified ' 
by clients, not Extension. Thus, In thls evaluatlonrobjetStives or goals stated bifore 
actual programs were conducted were not of concern and, in fact, did not exist. If 
they.hgd exlsted,tthey were not likely to be useful for identifying the additive 
benefits as perceived by participants. ; ^ 

The assumptions inhferent in this discussion mean that EMIS data are also too 
narrow;. they are limitedjo quantitative data— how many, when, how much time,, 
etc.— according to whaf^is pla^^ed as an educational objective, 

Thus, in'planning a similar evaluation, allow peopTejo name benefits that accrue 
over several years witheut regard to objectives that may or may not exist and are 
certainly irrelevant to most participants at survey tip^e. ^ ' 



8^ Evaluation should ioc^Sj oh recent (^'f 
j^aiO ^forts when trying |o assess 
impact owr tinie« 

This evaluation defined the program ai events and offerings occurring during 
the past 15 years (196M'975). This longer time frame wpi chosen to allovy evalu- ' 
.ation of multiple-year efforts. / \ . 

" The.15=year pertfed was appropriate to summarize a long-range Extension pro- 
gram and in relation to community resource.development projects (the leader ^ 
. survey). However, it was^usually too long a span for the general public res^ondente 
/ to recall meetings attended, topics covered, or benefits due to those meeting^* 

^- . Furtherrnore, Extension users of ev^luatior^data persisted in asking what per- 
' centage of people had contact during t\ie past single year% The more^recenj infor- 
mation se§med most relevant or users are simply used to thinkir^g of programs 
in terms of a single year; \ ' . \ 

4. ^ader identification incfuded retired county'board members, some of whom 
^ rfoTOTger felt involved in community affairs, had not used Extension, or vyere no^ 
interested in contributing opinions about Extension's impact Retired members ^ .^ 
could have been omitted' from the sample without loss of valuable inforrnati'on. 

In conclusion, five years is an appropriate time period. Questions. relative to 
participation Mn the psst year will also give useful data, " 



9« E]K^ct replication ofi this entire evalua- 
tion is unnecessary* 

The Shawano.evaluatipn was a pilot effort to learn about certairrkinds of evalu- 
ation. Reflection tells us that its cost, level of general ipforrn^tion, and other fac- 
tors makes duplication unwise. For example, some data were used and others 
weren't, and some methods of communicating results. were better than others. 
Some things worked/^and some.didn't " 

Several suggestions are: " " ^ , 

• The project's size should be related to: /ja) the.primary intended use . . . not 
every possible use; (b) the capability of thesysterh to develop ideas, design 
data cpllection, analyze, and report results; and Ic) the capacity and willingties 
^ of the audience or pai^ticipant systerri tp absorb and use informatipn wHife it 
is timely, " 



J ' •An early decision^ should be made on exact reasons for evaluation and on in- 
^ tended usage of data to be collected. People coor^dinating simitar evaluations 

are advised to set priowties systematically on data to be obtained and^nclude 
in surveys questions to get only those data. , * 

* The exact focus, goals, and procedures of another county impact evaluatio ■ 
should consider inclusion of data Sterns that were found useful by recipients 

of Shawano evaluation r'esuits. These relevant data itefjis are discussed 'further 
in Impact of Extension in S/jawano County: J, Usage and Appraisal. 

" . ^ • ■ -. j ' , . ■ 

' ' ^ / 

10. A program evaluation of this magni- 
tude requires an unusually large 
^ personnel investment before, during, 
and after data cblle(Dtion^ 

Eve'n if we presume this evaluation will not be repljcated, many types of resources 
are stilljj^eded in a smaller evaluation. Thousands of dollars, hundreds of hours, 
' and technical capabilities: to sample, collert data, analyze and communicate are 
, still needed in an evaluation of this size or if part of this evaluation. would be 
repeated. . ^ 

^ The Shawano eyaluatiori benefited by the cooperation of the University of 
Wisconsin^Extension's Division of^Program and Staff Development, the district 
, supervisor, the Shawano County Extension staH\excellent survey and ana 

specialists, supportive administration,, and particularly, cooperative and supportive 
local citizenry. These woulfj still bp-iT&eded in a smaller versiori of impact evalua- 
tion. * r - , 

■ " ' ' ^ " , ^ - ' \ ^ 

Specifically, a system contempfating a'similar, but smaller^ evaluation would need 
to allow for: , - ^ 

• A coordinator who will lead and manage the evaluation from beginning to 
end. ' . . * ^ 

. * Technically trained people in sampling, data collectioh, and analysis tasks. * : 

. Many hours of time for setting pfforfties on reasons for evaluation, data to be 
collected, and exact usages. 

• Persons in line supervisory roles who can ''bridge'' to other parts of the system 
when sharing data. 

* Many hours of time for interpreting, writing, and reportirig af4:er al I data have 
been collected. . ' . 
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Complex Statistical analysis is not 
neeessary for optimum utility and 
understanding of evidence. 

. ^ Data used for accountabiirty and program improvement may not need the soien- 
tificaily rlgoroui statistics commonly advocated by academicians.. Whether in or 
out of Extension, people want practical data related^t^ their concerns that help 
make choices now/are credible, and are informative. Such data may notTiave to 
come from a randomly drawn sample and a welhdesQribed population to be useful. 

, Data may not need sophisticated association or difference tests and inferential 

statistics if users want mere descriptions and percentages on how various groups 
answer key questions. . : , ^ : * 

In sum, provide findings in person or through papers in straightforward and 
easily read ways. Do not assume that all the hearers or readers understand or want 
statistical significance or scientific design. Rather, understanding and meaning are 
results of involvement, interest, credibility, and intelligibilrty, and not of mere 
volume or precision. 



la* Itoderstandihg data and deriving 
conclusions are as crucial as design 
and dlita : collecting. 

Evatuation is mor^ than tiata collection. Determining the value of a program 
' requires comparison^to some accepted standard. Interpretation of findings by 
* recipients is essenti^iito their accepta^ and use of data. Determining the real 
program value .and^^tt'rng interpretation by users occur in direct proportion to 
the- amount of /tinpej.given to those inds. 

This project, 'many others, devoted much time tp designing apjpropriate sur- 
veys and to sttnHa findings with faculty and administrators, but it also gave much ' 
tipne and use^ slyeral approachei to deter^ ^ . 

Prdjecft^tef m rnembers'identifled themes irt data, major conclusions, and posr 
sible ijTioiications for accpujit^bility or programming. The/e were shared in 
earlywri^iP reports. ^ ^ I ^ 

^ The prdjee^^team shared data at meetings. Audiences helped establish success 
"'^riTerii-a^d expressed what they viewed as the consequences or implications 
* of dpt^when discussing with others. ^ ^ 
• Evaiuattpn data shared througb the maii were accompanied by questions ask- 
ing.recipients to develop implications. ^ - , . , 

" . • Th4 district dLrecjorwho Initiated the evaluation^haired 
/ . ^ * disjnct/swe, and interhate levelsv wi^ 

" the milning erf these findings, ^'Wfl served js ah effective "bridge'' to other 
i partsto|^ Extension. 




In sum, (nany peaple helped disseminate and interpret results. The following 
conclusions can be made from interpretation experiences:* 

■ It is time-consuming and difficult to develop implications. ' ' , 

• Data have //ff/e value unless users help to develop conclusions ,and interpre= 
tatipns. . . 

• Indi'viduals and groups need stimulation through presentations and discussions 
to come up with implicationi. ; 

» Because of diver^|plues, needs, intferests, philosophies, and criteria by Exten- 
sion staff, interprerations are also dives^e; As such, eao^ interpretation is ^ 
valid in itself. Time is needed to develop and consider all interpretations. 

• More time must b§ given to ^earn about. developing interpretations and impli- 
cations of evaluations. , 



. Recipients of (evaluation data must 
identify their criteria for interpreta 
tim^-and-ref essence if data are to/ 
become meaningful. / 

' Evaluation has not occurred without comparison of WHAT IS (data) with an^ 
expressiorl of WHAT SHOULD BE (standard, criteria, desired level pf performance). 

I ■ 

Standard^ can be established by relying on; (a) similar situations; (b) authoritar' 
ian prescribed levels, such |s laws or what "experts" say; (c) research; (d) personal * 
beliefs and values; and (e) corhbinations of these ways. . 

When initiat evaluation data were released, E>jtension staff were asked to express 
i what they thought was an acceptable level of success befom they looked at tha - 
data^ More than 60 Extension staff and 10 county .board members participated in 
this "ietting-standards" procedure. ^ ^ ^ 

Specifically, the procedure asked individual staff 100 questions sucH as: "What 
perceotagaof thg, adults in Shawano County should have htd contact with Exten- 
sion in any way during the past 15 years?" and "What percentage of families ^ 
should havfe ^ad a child in 4=H at some time?" They did this ir\ group meetings " ^ 
and through the'rtiafL The procedure alerted them to soon-to-be revealed data and 
helped them get personally ir^olved in the data. - ; . \^ 

During group'meetings^ infornlal discussion of expectations became'part of the 
data presentation. Some people sbared what they thought shoufd be the ^percentage ^ 
of participation and why. On theobasis of a different perception of the county, 
-others suggested a different leve^ Many, people learned more about the county. 
Many clarified their beliefs about success. Group consensus nfiay have been useful 
and could have been reached, but in these situations it was not considered necessary. 



■ . 19 ' 

c ■■ . ■ ' ■ 

Setting standards within group meetings was much more effective than by mail. 
Many minds working together contributed various ideas and bonditions for "success/ 
failure/" For instance, individuals whd consistently set a standard of 100% partic- 
ipation>(because "everyone ought to have contact with us") were encouraged to 
consider that: (a) Extension may not have programs to meet every need, (b) -some 
people feel they don't need Extension, or (c) those who have never heard of Exten- 
sion cannot use its prbgrams. ' ' . ' 

On the other hand, those. who'set very low expectations ("less than 1% of a 
county population using Extension is enough for success") were reminded of the * 
13 program areas and various means'of reaching a broad range of Clientele, '] 

' In summary, we strongly recommend that future total impact evaluations set 
standards at or near the time of data collection and reporting. Specific procedures 
can vary, but the general process is a must for several reasons: 

• Data are , meaningless to ^people without their own reference points for inisr- 
pretation. Data become more purposefuL " ^ ^ 

• All multiyear Extension programs involving adults who have their Own needs- 
have evolved or changed drastically from an original goal if they existed at all, 
and thus originargoals ar^ not useful as a reference point 

• Users of data' are not likely to be the same people who originally set the goals 
and thus do not automatically accept the goals stated 5-15 years ago as valua- ' 
ble. , / . 

• Cross-discipline or multiple-unit program evaluations are less likely to have 
preprogram goals, to which data can be compared, 

• A criterion that an Extension pferson sets becomes a commitment. 

• The setting of criteria is a learning process for Extension professionals. 



I4. A yarlety of titnely, short, simple^ 
attractively visualized techniques to 
conimuiilcate evidence gets accept - 
ance# j 

The Shawano evaluation shared findings in many ways: (a) long narrative written 
materialsr (b) short summary papers; (c) brochures highlighting findings on a partic- 
ular topic and announcing the availability of longer papersi (d) professionally pre- 
pared audiCHvisual media such as slides and transparencies; (e) group meetings with . 
Shawano County agents and leadep, statewide, district, and county Extension fac^ 
ulty, ES-USDA (now SEA^Extension) administration. North Central districtydirectors' 
association, and the national advisory committee for the project; and ff) the series 
of three major reports, / 

Preliminary dfafti went to selected faculty f0r';reactloh. . Finished repprte were 
given a wider di^ribution, both vvithin Wisconsin and out-ofst^^ 
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Several problenqt \A;ere encountered. It was difficult to get the right information 
to the right peopie when they wanted it! On the other hand, some people still are 
in no hurry to consider the possible implications from this evaluation. 

Soma individuals received massive amounts of papfer=s^me in draft form to.be 
^ reread in a revised form. /So many titles and so muc-h sim!l|Fity blurreti the re^l 
distinctiveness of each pFece of information. Thus, "overkill". happened in some ^ 
cases, - |. ' ' 

We concluded that: ' ' 

^ * Potential users must be personally involved. A personal involvement is more 

easily secured through pleasant, informal sharing, where data are easily underr 
stood. Consider the time available to audiences, their commitment to^beirig 
involved. Reinforce points with words and visuals. Don't ask people to wade 
through 64 pages ffrst, then send them a ^page highlight 3 weeks later. 
Arrange the reporting of evlde/ice according \q how people learn: awareness, 
involvement, acceptance, practice, and change. * . ^ ■ 

" • TvyO'by^two slides were less conducive togroup sharing and discussion. Invest 
in overhead transparencies to aid the process of presenting percentages and 
tables. . Transparencies help organize information according to themes, major 
findings, major conclusitons, clientele or program specific information,: 
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What Next? 



As the previous conclusions have imph'ed, the impact evaluation of Shawano 
CDunty's total Extension program has already provided a better picture of B)cten-£ / 
sion's value to the county and its people.VTh^rnethods may thus hay&^ome utility 
in other counties and states for purposes of accountability^ which, without question, 
is continuing to increase in importancp. Ip looking to answers on how to deal ^vfth 
these accountability pressures, be creative and imaginative; do not feel that old 
procedures are necessarily the besj. Consider threeimethods: (1) Shawano evalua- 
tion methods, {2) ones you've used, and (3) new, creative ones you've yet to try. 



( 
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Additional Materials 

. Setting Stmdards tn Evaluation, A Methodology 

* PrQcedure used^o elicit lev^ of participation standards from Extension personnel 
reviewing Shawano Impact data, 

Cpnc&p t$ an d Procedur&s of the Shawan o E valua don Prpjec t 

A detailed reference to desigrTlnd methodology used in the project, 

^.Developing a Survey of the Combined Impact of Extension ' ' 

Simplified survey for limited resource situftions. , ^ 

Supplemental Approach to Questions in a Combined J mpact Study 
J^isting of alternative questions for a combiDed impact study.: 

., De^lopJng a Survey that Combines Several Separate Studies 

' ' ' Discussjon of how, when and why of combining two or more studies in one survey^ 

' ^ .■ ^ _ ^ ^ ^ ' 

An Approach to Examining Farmers' Use of Extension 

. . Questions used in^the Shawano Project plus suggiwions, . , 

An Approach to Examining Impact of 4- H ' 

Questions used in the S^hawan^ Project plus suggestions. , " 

An Approach to Examining the Impact of Extension Methods * 

Questions used and organization of the Shawano Projict, ■ ' ^ ^ 

. ^ : . f / , ^. \ . ^ ' 

^ An Approach to Examining Impact of Home, Economics Programs ^ ^ 

Questions used in the Shawano Project plus suggestions. 

Data Patterns and Mean ingsi^ , , u 

ixaroples of how data were handled affir they were secured. 

^> ■ . ' ' ^ 

Telephone SL/rveys as an Extj&nsion Tool ' ^ . \^ 

Based on experiences in addition to the Shawano Project, this piece deals with 

specifics for preparing a teFephone interview schedule. 

Time Efficient Ways of Following Up On Programs 

Showing how the Benneff categories of evidence can be used in several situations, ' . ' 
this piece uses the Shawano Project as an example. , 

Survey Schedules: A, C, Leader 

A & B are lengthy sets of questions for respondents with considerable contact with 
Extension; C is a shortened version. The leader schedule parallels other schedules 
in some ways. = * v ' 
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Major Report in this series: Imj^Gt of Ixtenifpn In Shawano County 

1, Conclusions & Implications * 

2, Methodology ' " 

3, Usage & Appraisal ■ '■ 



MAJOR RiPORTS are available at no charge to anyone. Other materials will be 
reproduced as requested, at no charge to Wisconsin-Extension personneL Please 
request a price list for materials to other agencies and states. 

Or'&er through: ' ' 



Program and Staff Development 
Unjviriity of yyisaoniin-Extension 
601 iKtfnsiOn'Buildjng 
432 North Lake Street 
IVIadiion, Wl 53706 
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